Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	Nemo_bis	Date:	Nov 3, 2013 3:23am
Forum:	web	Subject:	Re: Allow IA to index, prevent search engines

On a standard website, the easiest approach would be to put those you want to be archived in a different directory, use robots.txt instead of meta tags and whitelist the Wayback machine for that directory (though I don't know if Allow: is accepted by the machine), User-agent: ia_archiver according to https://archive.org/about/exclude.php